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(57) Abstract: The invention relates to a method and apparatus 
for delivering a test, such a test to a candidate. The invention also 
concerns an apparatus and method of assessing the relative cogni- 
tive ability of a candidate. The invention also concerns a method 
of assessing the progress of a candidate in response to a learning 
program. The test, assessing or learning program comprise de- 
livering a s^es of questions type to a candidate, the questions 
increasing in difficulty as the candidate successfully completes 
easier questions, the questions being delivered to the candidate 
in ranges. The questions may be "Swaps" type questions. The 
candidates may provide their confidence that their answer to any 
question is correct A quantifier may be used to assess the candi- 
dates progress and/or cognitive skills. 
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A Method of Delivering a Test to a Candidate 

T gp^nical Field 

The invention rdates to a method of delivmng a test and in particular to a 
5 method of delivering a test to a candidate. 

The invention has bera developed primarily for testing the relative cognitive 
development of cUldren and will be desoibed hereinafter with refermce to that 
application* It will be appreciated, however, that the invention is not limited to that 
particular field of use and is also iq>plicable to testing the cognitive abilities of adults, 
10 including those who have experienced a loss of cognitive ability due to a physical 
accident or a mental illness. The invention is also applicable to assessing the progress 
of a individual with learning difficulties using a learning program. 

16 It has been known to test the cognitive abilities of individuals to gain an 

indication of the suitability of that individual for a given role in a workplace. This has 
been qpplied to reciuitmmt programs for businesses and other organisations. It has 
also been known to apply confidence indicators to these tests tx> further refine the data 
provided. 

20 However, the known testing is difScult to accurately qn>ly to children and those 

witii learning difficulties. That is, tiie testing foimat is unsuitable for individuals of 
immature cognitive functioning as those individuals are presented with items that are 
beyond their memoiy operating potential. Moreov^, the format has no regard for any 
progressive changes in ability to deal with cognitive complexity that arc associated with 

25 maturation and witii old age. As such, tiie current vision of the test does not afifoid a 
ready way to track cognitive development/degeneration. 

Any discDSsLon of the prior art throughout the specification should in no way be 
considered as an admission that such prior art is widely known or forms part of 
common genial knowledge in the field. 

30 

According to a first aspect of the invention there is provided a method of 
delivering a test to a candidate, the test having a plurality of questions' that are chosen 
from a plurality of question types ranging fix>m an easiest question type to a hardest 
35 question type, the method comprising the sequential steps ofi 
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delivering to fhe candidate a question selected from the easiest question type 
and, tiiereafier» delivering to tbe candidate one or more questions from the same 
question type; 

delivering to the candidate a question selected from another of the question 
5 types and, thereafter, deliyeriDg to the candidate one or more questions from the same 
question type; 

wherein at least one question is a **Swaps** question, the ''Swaps*' question 
conqnising displaying a plurality of images to the candidatB in a predetermined s^>atial 
relationship and an instruction sequence associated with the images for indicating to the 
10 candidate the required reorganisation of the spatial relationship of the images. 

The test may include question types 1, 2, ...» o, where type 1 is the easiest 
question type and type n is the hardest question type and the method comprises 
sequentiaUy deUvmng to the candidate the questions of question type 1, question type 
2, ... and question type n. Optionally, all the questions of a given question type are 
16 sequentially delivered to fiie candidate. The candidate may only be delivered a 
question when a preceding question in the sequence has been answered. 

Optionally, upon answ^ing a question* the candidate is asked to indicate his or 
her confidence that the answer to the question was coirect. The candidate may be asked 
to Indicate his or her confidence that the answer to the question was wltiUn one of two 
20 or more confidence bands. There may be ^ confidence bands. 

The test mi^ be delivered by a testing device that comprises a display for 
visually displaying a question to the candidate and an input device for allowing the 
candidate to answer the question. The testing device may comprise a processor for 
. driving the display and for beiiig responsive to the input device for determining the 
25 answer provided by the candidate. Optionally, the input device con^rises a k^board. 
In other embodiments the input device comprises a pointer device such as a mouse, a 
track ball or a touch pad. In some embodiments, the input device coinprises a 
combination of the keyboard and the pointer device. 

Optionally, the test is contained as question data on a storage medium and the 
30 processor selectively accesses the question data to deliver the questions sequentially. 
The storage medium may also comprise sequence data r^resentative of ttie sequence in 
which the question data is accessed by the processor. Even more preferably, the storage 
medium is controlled by a server ttiat is remote fiom the processor and which is 
responsive to predetermined input from the processor for allowing the test to be 
35 delivered to the candidate. 
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In one embodiment the test is deUvered in HTMI^ while in embodiments 
it is delivmd as XML. However, in dtemative embodiments, the storage medium is a 
compact disc or other portable stora^ media that is physically provided to ttie 
candidate for interaction with the processor, hi other embodiments, the question data 
6 and sequence data are jirovidcd electronically as an executable 61e to the processor 
and/or the candidate. 

Optionally, the "Swaps" questions axe selected from the above question types, 
whidi indude: 

1. The easiest question type or Level 1 - die candidate is instructed to memorise 
10 the relative spatial order of the three images and, once the candidate indicates he 

or she is ready, the test progresses to the next step and requests the candidate to 
indicate that order from memory. In this Level 1 case there are no swaps per se. 

2. Question type 2 or Level 2 -the candidate is provided with three linages having 
a pred^ermined relative spatial order and is instructed to complete a single 

15 swap of images. When the dmdidate indicates tiiat he or she is ready, the test 

progresses to the next st^ and requests the candidate to indicate the revised 
order fiom memory. For example, lbs candidate is, in some cases, asked to 
sw^ the second and the third image. 

3. Question type 3 or Level 3 — this is similar to the previous question type, with 
20 the exception fhdX ttie candidate is requested to complete two 8ws(»s of the 

images Aat have been provided. For example the instructions in one 
embodiment are to swap image 2 and image 3, tiien $w^ image 1 and image 3. 
Once the candidate indicates that they are ready, the test progresses to the next 
step and requests the candidate to indicate fttc revised order firom memory. 

26 4. Question type 4 or Level 4 - as with &e prior type excqst that die candidate 
completes three swaps. In one embodiment the instructions include, for 
example, to swap image 2 and Image 3, dien swap tma^ 1 and image 3» and 
then swap image 1 and image 2. 
S. The hardest question type or Level 5 - the candidate is instructed to complete 

30 few sw^s. For example, in one CTibodiment, the Instructions direct that the 

candidate swap image 2 and image 3. then swap image 1 and image 3, then 
swap image 1 and imiage 2, and then swap image 1 and Image 3. 
Optionally, there may be more question types with die additional question types 
requiring a larger number of swaps. 

35 The candidate may be provided with a plurality of alternatives firom which the 

answer is selected* However, in other embodiments the candidate clicks and drags the 
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images to fhe locations that are believed to be coireot Questions of kind are 
described in more detail in the earlier filed International Patent Application 
PCT/AU99/00816, the disclosure of which is incoxporated herein by way of ofoss 
reference. 

5 Optionally, the time taken for the candidate to read and answer a question is 

recorded Optionally, fhe tinie taken by the candidate to select a confidence band is 
also recorded. 

According to a second aspect of tiie invention tiiexe is provided an i^paratus fi»r 
ddiveiing a test to a candidate^ the test having a plurality of questions that are chosen 
10 fiom a plurality of question types ranging fiom an c^est question type to a hardest 
question type, the apparatus comprising: 

a display for sequentially displaying to the candidate one or more questions 
selected fiom the easi^t question type; 

it^mt means for allowing the candidate to answer the questions and for 
15 triggering die display to display to the candidate the next question in tiie sequence or, if 
at die end of the sequence, one or more questions 6nom fhe next hardest question type; 

wherein at least one question is a "Swaps** question, the "Swi^** question 
comprising a plurality of images displayed in a predet^mincd spatial relationship and 
an instruction sequence associated with the images fbr hidicating to the candidate fhe 
20 required reorganisation of tiie spatial relationship of fhe images. 

According to a tiiird aspect of fhe invention tiiere is provided a method of 
assessing the relative cognitive ability of a candidate, fhe assessment having a plurality 
of questions that are chosen fix>m a plurality of question types ranging progressively 
fi-om an easiest question type to a hardest question type, the method comprising the 
25 sequential steps ofi 

ddivering to fhe candidate a question selected fimn fhe easiest question typ^ 
allowing the candidate to answer fhe question and, thereafter, delivering to the 
candidate sequentially one or more questions iBmn fhe same question type; 

deteraiining a quantifier in response to the answers provided by the candidate to 
30 the questions of the easiest question type; 

if the quantifier is above a predetermined ttu-eshold for delivering to the 
candidate a question selected fiom another of tiie question types, delivering to the 
candidate a question selected fimn fiie said another of the question types and, tiiereaft^, 
delivering to fiie candidate one or more questions fitim the same question type; and 
35 deriving an assessment of the relative cognitive ability of a candidate using die 

quantifier and fhe question types for which answers have been provided. 
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Optionally, the said another of the question types is the next in the progressive 
range of question types. 

The method may also comprise detmninii^ a second quantifier in response to 
the answers provided by the candidate to the question of the said another question 

S types. If the second quantifier is above a predetermined second threshold for delivering 
to the candidate a question selected from the next question type in the progression of 
question types and, the method may comprise the steps of delivering to the candidate a 
question selected from the next question type and dLereafter» delivering to the candidate 
one or more questions from the same question type. That is» the candidate is not 

1 0 presented witfi a question from a particular question type unless he or she has answerod 
correctly sufficient of the questions in Ae preceding question types. In some 
embodiments, the quantifier is detemiined from the answers of selected ones of the 
questions of die question type. 

According to a fourth aspect of ttie invention there is provided an apparatus for 

1 5 assessing die rdalive cognitive ability of a candidate, die assessment having a plurality 
of questions that are chosen from a pluzality of question types rangmg progressively 
fit>m an easiest question type to a hazdest question type, die ^aratus comprising 

a processor for driving a display to deliver to the candidate a question selected 
from the easiest question type; 

20 an input de^ce for allowing the candidate to answer the question^ wherein the 

processor is responsive to the . answer being inovided to subsequendy ddiver to the 
candidate sequentially one or more questions from the same question type; 

a calculation module being responsive to the answers provided by die candidate 
to the questions of the easiest question type for determining a quantifier, wherein die 

25 processor is responsive to die quantifier being above a predetermined threshold for 
driving the display to deliver to the candidate a question selected from another of the 
question types and, thereafier, detivering to the candidate one or more questions firom 
the same question type; and 

an assessment module for bdng responsive to the quantifier and the question 

30 types for which answm have been provided for deriving an assessment of the relative 
cognitive ability of a candidate. 

According to a fifth aspect of the invention there is provided a mediod of 
assessing the progress of a candidate in response to a learning program, the assessment 
having a plurality of questions diat are chosen fix>m a plurality of question types 

35 ranging progressively from an easiest question type to a hardest question type, the 
method comprising the sequential steps o£ 
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delivoing to the cataiclidate a question selected fiom the easiest question type; 

allowing the candidate to answer the qii^on and, ttieieafter, delivering to the 
candidate sequentially one or more questions from the same question type in response 
to the preceding question being answered; 
6 detennining a quantifier in response to the answers provided by the candidate to 

the qu^tions of the easiest question type; 

if the quantifier is above a predetemiined threshold for delivering to the 
candidate a question selected fiom another of ttie question tyP^9> ddivering to the 
candidate a question selected fiom the said another of die question 9pes and, thereafter, 
10 deUvering to the candidate one or more questions firom the same question type; and 

diving an assessment of the relative cognitive ability of a candidate using the 
quantifier and the question types for vMch answers have been provided. 

Optionally, at least one question of the third, fourth and fifth aspect of the 
invention is a "Swaps" question. 
15 In one embodiment, the quantifier is deterauned upon a weighted sum of any 

one or more of 

correctness of the answers provided by the candidate; 

time taken to provide those answers; and 

the relationship between the correctness of the answers and confidence indicated 
20 by die candidate that his or her answos to die question was correct 

Brief Descriotion of the I>rawinpa 

Preferred embodiments of the invention will now be described, by way of 
example only, with reference to the accompanying drawings in which: 
25 Figure 1 is a sdiematic view of an iq)paratus for delivering a test in accordance 

with the invAtion; and 

Figure 2 is a flow chart of aprefeired mediod of using the q>paiatus of Figure 1. 

Detailed Description of the Drawings 

30 Refming to the drawings there is illustrated a desktop completer 1 having a 

desktop unit 2 in which is housed one or more processors, RAM, ROM, bard drives, a 
networlc card, othor cards and devices and die controlling software. None of Oiese 
hardware of software compon^ts are specifically iUustrated and it will be understood 
by those skilled in the art fliat many options and combinations of these components are 

35 available* 
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Computer 1 also includes a video display unit (VDU) 3 that interfiEU^ with unit 
2 to display to a candidate selected still and/or video images. An input device, in the 
foim of a keyboard 4 and a mouse 5 allow die candidate to enter data and/or to answer 
the questions that are presented to Aem, as will be described in more detail below. 
5 Desktop unit 2 is connected to a network, which in this case is a LAN 6, via a 

network card (not shown) and cabling 7. The LAN makes use of one of a variety of 
network protocols and is delivered via a networic server (not shown)* In some 
embodiments, the functions of the netw^ork server are shared amongst a plurality of 
intenxmnected servers. 

10 A test delivery server 8 is connected to LAN 6 and has access to a storage 

medium in fiie form of a bard drive 9. This hard drive includes a database of questions 
that are graded into a plurality of question types ranging from an easiest question type 
to a hardest question type. The database also includes data indicative of the sequence 
in which the questions are to be asked and other data for allowing a detemiination as to 

15 ydieAer or not the sequence is to be progressed. This is set out below in mm detail. 

Initially,^ the candidate turns con4>ut^ 1 on and logs into LAN 6 utilising the 
designated user name and password or passwords. In this case, the candidate is due for 
an assessm^ and is provided a pmmpt by the network that this is so. Preferably, the 
prompt is in the form of an email including a hypeitext link that tbe candidate is able to 

20 click using mouse S so that the first page of die test is delivered. However, in other 
embodiments, the LAN recognises the user and that an assessment is due and 
immediately initiates the test 

In fins embodiment, the test is delivered in HTML pages (hat are obtained fiom 
hard drive 9 and sequentially delivered to desktop unit 2 via server 8 and LAN 7 and 

25 displayed on display 3 to the candidate. In otfa^r embodiments^ different languages or 
formats are utilised. In some embodiments the delivery language is XML, alfliough 
other alternatives are also available. 

In oHicT embodiments, use is made of other programming languages. For 
example, in an 'internet" delivered embodiment, flie programming is written in the Java 

30 language, and the tests as discrete packages or Java Servlets that are transferred to the 
endp-users' computers (the respective candidates' computers) to allow 'Veal time" 
measuTCTient of reaction time/speed of re^onding. 

Once the candidate has selected the rrtevant hypertext link in the email referred 
to above, the test is conmienced and follows the steps set out in gr^bical form in 

as Figure 2. The candidate is initially presented with a HTML page that requests 
confinnation of his or her personal details and the ruiture of the assessment being 
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conducted. Qnoe fhis confinnatioii has been provided, the candidate is delivered a 
subsequent page that inchides an introduction to the test and some instructions a$ to 
how the question should be answered. Preferably also» the candidate is provided with 
the choice of completing one or more sample questions so as to become more &miliar 
5 with the question fbmiat. 

Before progressing further throug^L the process of Figure 2, it is benefida] to 
look more closely at the format of the test In this embodiment, the test includes a 
plurality of questions that are selected fiom the questions stored on the database. These 
questions are all categorised with a question type and are designated as question type 1. 

1 0 question type 2, . . question type n. Where type 1 is the easiest question type and type 
n is the hardest question type and the intervening types are graded progressively 
between the two eKizemes. That is, as the question type inareases, so to does the 
difficulty of the questions contained within that type. 

In the fidd of cognitive testing, such as Aat witti wbidhi the ptiaierred 

15 embodiments is concerned, it is known to ask questions that are referred to as ''Swaps" 
questions. This form of question is one of tiie forms of questions set out in 
International application no. PCT/AU99/00S16 which was filed in the name of the 
present applicant The disclosure contained within that application is incozporated 
h^ein by way of cross refi»ence. 

20 As used in this embodimmt, die '^Swaps*^ questions include a plurality of 

images, usually three, that are displayed to the candidate in a predetermined spatial 
relationship. Included under the images is an instmction sequence indicating to the 
candidate the required spatial reorganisation of those images. This reorganisation is to 
be performed mentally by the candidate who, once satisfied with the information at 

25 handy has the question removed and is asked to choose, as an answer to the question, 
one .of a plurality of spatial combinations of the images that are presented as bdng 
possible answm. The tinie taken to cognise the question and chose the answer is 
timed« Moreover^ after completing the question, the candidate is asked to quantify his 
or her confidence that the answer was correct. Tliis quantification is within one of six 

30 bands, those beiiig 0% to 17%, 17% to 33%, 33% to 50%^ 50% to 67%, 67% to 83% 
and 83% to 100%. Ihotherembodunents a different number of bands are used. Instill 
further embodiments, ttie candidate is asked to nominate a perc»tage confidence* As 
discussed in the abovementioned PCT patent application, there are embodiments whm 
a confidence indication is sought from the candidate aftst each question is asked, while 

33 in other embodiments, that indication is only sought for less than all the questions. 
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In Hda embodimepti the images used in die "Sw^" questions are clearly 
visually distinct Examples of ^diree images are a house, a car and a bird. Another 
example is a hammer, a dog and a balloon. As will be aiqpreeiated finom the teaching 
herein, many other combinations are possible. 
5 In one embodiment, the starting combination of the images is always the same. 

So, for the examples of the images being that of a house, a car and a bird, ttie candidate 
is always provided with the same spatial arrangement of those Images widi each 
question. In ofh^ embodiments, however, the starting combination is varied 
throughout the test That is» even within each question type, the qxiestions will include 
10 dififerent combinations of images that are presented to the candidate with the writtm 
instructions. 

In this embodiment* dxe starting combination is varied through the test both in 
the actual images used and the spatial arrangement of those images. While these 
factors are predetermined for each question> so as to provide equivalence for difference 
15 candidates, in other embodiments the determination is random so. that different 
candidates will be presented with different combinations of images. 

The test of the preferred embodiment has five question types, which include: 

1. The easiest question type or Level 1 - the candidate is instructed to memorise 
the relative spatial order of the three images and, nnce tlie <?mdidatf^ ^nrfi^^^teg he 

20 or she is ready, the test progresses to llie next stqp and requests the candidate to 

indicate that order from memory. In tfiis Level 1 case tfam are no swq>s perse. 

2. Question type 2 or Level 2 - the candidate is provided with three images having 
a predetermined relative spatial order and is instructed to complete a single 
swap of images. When the candidate indicates that he or she is ready, the test 

25 progresses to the next step and requests the candidate to indicate the revised 

order fiom monory. For example^ the candidate is, in some cases, asted to 
swap the second and the third image. The convention being, that tiie images are 
arranged in a linear horizontal array and, progressing fiom left to right, the 
images are referred to as the first, the second and the third images respectively. 

30 Alternatively, they are referred to as image 1, image 2 and image 3. In some 

embodiments, the images are accompanied by a corresponding reference 
. numeral to assist the candidate in understanding this referencing convention. 

3. Question type 3 or Level 3 - this is similar to the pfBvious question type, with 
the exception that the candidate is requested to complete two swq^s of the 

35 images that have been provided. For example, the instructions in one 

embodiment are to swap image 2 and image 3, then swap image 1 and image 3. 
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Once the candidate indicates . that they are ready^ fhe test progresses to the next 
step and requests the candidate to indicate the revised order fionx monoiy. 

4. Question type 4 or Levd 4 — as with the prior type except that the candidate 
completes tibree swaps. In one embodiment the instructions include, for 

5 example, to swap image 2 and image 3, then swap image 1 and image 3. and 

then swBp image 1 and image 2. 

5, Tbe hardest question type or Level 5 - the candidate is instructed to con^lete 
four swdps. For example^ in one onbodiment, tiie instructions direct that die 
candidate sw^ image 2 and image 3, then swap image 1 and imago 3, then 

10 swap image 1 and image 2, and then swap image 1 and image 3. 

Optionally, th^ may be more question types with the additional question types 
requiring a larger number of sw^s. 

Once die candidate bas reviewed the images and the associated instructions 
concerning the swaps that are to be cognised, he or she uses mouse S to activate a 

15 tQ^p^text link ttiat deliveis to the candidate a plurality of alternatives from whicii flie 
answer is to be selected. That is, the questions are in Ae form of mxiltiple-^olce 
questions. Importantly, the candidate is not now able to access both the question and 
the initial spatial arrangement of images upon which the question is based This 
requires the candidate to retain information, in the short temi, of flie spatial order of ttie 

20 . 'Wapped** images. In the case of web delivered test, this is achieved by disabling the 
history iunctioa and ^'Back*' button of the web browser. In other embodiments 
altonative means are relied upon. 

In other CTibodiments, once the candidate indicates that he or she has read the 
required instructions, the next page is delivered, but this does not present the candidate 

26 with a choice of answer. Rather, Oie candidate is presented, with the images in the 
original relationship and required to use the mouse to sequentially designate which of 
the images is in ttie left most, tiie middle and the rigjht most positions. For non-web* . 
based delivery making use of this mode of answer, the us^ is required to **click and 
drag" the images - which are present in tbe initial spatial relationship - to the locations 

30 that are believed to accord with that directed by ttie instructions. 

The time taken for the candidate to read and answer a question is recorded. 
More preferably, the time taken by die candidate to select a confidence band is also 
recorded. The first' is used to provide an Indication of the level of candidate's 
confidence that he or she correctly answered the questions concerned, while the second 

35 provides an indication of fhe general confidence of the candidate. 
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Roferriag to Figure 2, once die user has entered his or her details, server 8 is 
provided with that information and configures flie test to be delivered^ in the event that 
that is required for that candidate. For example, in some embodiments the candidate is 
attempted one in a sales of tests to provide an ongoing assessment of both the 
5 candidate's cognitive abilities and any progress or improvement that is being made in 
response to a learning program that the candidate is being subjected to. In diis case, 
server 8 is responsive to the identity information provided by the candidate to prepare 
to deliver the desired test in the series of test being undertake. That is, die test regime 
is not only responsive to thie candidate, but also the results of earlier tests taken by the 

1 0 candidate and the resxilts of those tests, 

Once servo: 8 has configured the test and the proposed questions to be presented 
to the candidate, those questions axe mmg^d in groups of question types such as the 
Level 1, Level 2, Levd 3, Level 4 and Levd S question types referred to above. Each 
of these question types includes usually five or more questions of that type. That ls» a 

16 typical test includes twenty five questions in total. Howev^> in other embodiments it 
more or less questions of each question type are used. Moreover, in further 
embodiments, there are not necessarily the same number of questions in each question 
type.. 

"While the accural of die test will be theoretically improved witii the greater 
20 number of questions asked, there is a practical trade off due to the attention span and 
concentration levels that are able to be provided by the candidate* For young children 
and candidates with learning disabilities, relatively few questions are used, while for 
adults it is possible to gain more accuracy through the use of a greater number of 
questions. 

25 Server 8 also designates each question in a given question type with a ref«rence 

that determines the order that fbe questi<m will be presented to the candidate. For 
example, in this embodiment, the Level 1 question type includes &ur "Sw^s" 
questions, which are refanence as 1 , 2, 3 and 4 respectively. This referoice coiresponds 
to the variable '*x" in Figure 2, while ttie question type corresponds to the variable '*n". 

30 Initially^ both x and n are set to 1^ and are incremented by 1 in ac^rdance with 

the flow chart of Figure 2. As will be apparent to a skilled addressee, fiom the teadung 
her^, X and n are integers, where 1 ^x^Xfi^andl riSn^n^^^). In this embodiment, 
x^B^ » 4 ftir the question type refinred to as Level 1» while n^„^ « 5, in that there are 
five question types. In other embodiments both x^^^j and have other values. 

35 Server 8 presents the candidate with question 1 of the Level 1 question type. 

Once the caxididate is satisfied with the question, use is made of the mouse to indicate 
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that this is the ca$e. For example, far die wd>-delivered test of the present 
embodiment, ttie candidate provided that indication by manoeuvring mouse 5 so that 
the cursor displayed on VDU 3 overlies a clearly labelled 'Troceed** button on the web 
page. One of the buttons on mouse S is depressed by the candidate to click the 

5 '^Proceed^' button. Serv^ 8 delivers a subsequent web page providing the choices fix>m 
whldi &e candidate must select ihe answer to the question. Again, use is made of 
mouse S to dick on the selected choice. Server 8 is responsive to tiie click to not only 
gather the relevant information as to the answer provided and (he time taken by the 
candidate, but also to deliver a subsequent page requesting a confidence indicator from 

10 the candidate. In this embodiment, there are six confidence bands that are graphically 
represented on VDU 3. The candidate selects one of the bands through use of mouse 5. 

With die first question bemg answered and a confidence indicator obtained, 
senr^ 8 increments x by I and determines whedier the new vahie for x exceeds x^^^^ 
If not, the test progresses to ask the next question in that same question type. This loop 

15 - referred to as the first loop — is continued until x « x^.^. That is, all the questions of 
the easiest question type are presrated, and answers obtained, prior to any other 
questions of a dififermt - and in this case more challenging - question type are 
presented. 

Tlie test also includes a ''time ouf provision, in that if the candidate does not 
20 answer the question within a in^edetamined period, die test will signal that the time 
available to answer the question has li^sed. Server 8 is made aware of the time out, 
increments x by 1, and then proceeds with the steps that would have otherwise bem 
followed 

If; once incremented, x > x^ax> tt^^ fhm are no further questions of that 
25 question type to be presented to flie candidate fixr this particular assessment That being 
the case, die test branches from the first loop into a second loop. This second loop 
includes the first step of determining whether n » n(.i^. If yes, then the test is complete 
and the candidate is informed accordingly. If no, server 8 deteroiines a quantifier for 
die question type. This quantifier is referred to as Q^. For this embodiment there will 
30 be a maximum of four quantifiers calculated, Qi, Q^, Q), and Q4. A quantifier is not 
required for the Level S question type, as will be explained below. However, in some 
embodiments Q5 is calculated for use in subsequent assessments undertaken by the 
candidate. 

In this embodiment the quantifier is calculated by server 8 based upon a 
35 weighted sum of. the coirectness of the answers provided by the candidate; the time 
taken to provide those answers; and the relationship between the correctness of the 
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answers and Qie confid^ice indicators. In other embodiments, the quantifier is 
calculated in response to less fiictois or» in further embodiment, other factors. If the 
quantifier does not exceed the tfar^old, the test ends. 

If the quantifier, once calculated for a given question type, exceeds a 
5 predetemiined threshold for that question type, n is incremented by 1 and x is set to 0. 
That is, server 8 progresses to the next question type, as all fhd question of the 
preceding type have aheady beoi pxesented. The process then returns to the first loop 
until all the questions of the new question type are presmted and answered. V/hen this 
has occurred, the process returns to die second loop. When all the questions to be 

1 0 presented have been presented, the test ends by branching to the *^d Tesf ' box. 

As referred to above, the question types are presented in order of difficulty. 
Accordingly, a candidate is only required to answer the more difficult questions only 
once he or she has dononstnited that the easier questions are able to be sufSdenfly 
cognised and correctly answered. Moreover, the threshold for the quantifiers changes, 

15 in this embodiment, for the diffomt question types and the candidate. That is, fi)r the 
higher level question types, the quantifier is lower, as more time is allowed for the 
candidate to answer these questions. 

In other embodiments it is known to use more or less than five levels of 
questions. In practice, however, it has been found that five levels is a good balance, 

20 particularly for yoimg^ candidates. 

In other embodiments the confidence of the candidate may be captured in a 
different way, such as typing in the specific percentage or any other method that 
enables fiie infiomation to be ccqptured. 

All embodiments, regardless of the nmnber of question levds, present those 

25 questions in a systematic and structirod order, progressing fi!om the easiest ^e to the 
hardest type. This allows the candidate to only have to answer as many questions as are 
relevant, and not have to be unnecessarily preoccupied with answering questions that 
axe simply beyond their cognitive ability at that time. Hie nature of the test also allows 
it to be regularly reused, particularly in cases where a candidate does not progress 

30 through all the available levels of questions* 

In tiie embodiment described above, the test automatically advances and 
concludes in accordance with the answers provided by the candidate* In other 
embodiments, however, the administrator of the test has the ability to intervene and 
progress the test or conclude fiie test, notwithstanding that this would not have occurred 

35 othowise. That is, the administrHtor is provided with teal time access to the test results 
so that in situ control by the administmtor is allowed. 
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The prefenred embodiment of the invention includes fhe following advantafies: 
h It is ai»plicable to a wide range of age groups as the complenty of the test 
changes b^een levels. That is, test takers range fiom veyy yoxing children to 
senior age adults. 

5 2. It is used to determine a level of cognitive ftmctioning (by determining Qio 
complexity the test taker efiGactively deals with) in comparison to test takers of 
tiieir own broad age group. As such, delays in functioning, or fimctioniiig over 
and above expected levels, are more readily mapped throu^ a single test. 
3* Improvements/deterioration in cognitive fimctioning are readily determined 

10 overtime. 

Applications of the preferred embodiments include: 

1. Assessing the cognitive development of duldren; 

2. Assessing the inogress of an Injured adult in a cognitive rehabilitation program. 
15 3. Assessing tiie cbgnitive effects of a pharmaceutical on a candidate. 

4. Assessing the cognitive effects of aging and disease. 

The embodiment described in detail above utilises an input device in the form of 
a desk top computer having a VDU, a mouse and a k^board, amongst other things. Izi 
some embodiments, however, use is noiade of a toudi screen. In oth^ embodiments, 
20 use is made of voice recognition software and the associated microphones to allow the 
u$en» to input their respecdve responses to fhe questions. 

It will be appreciated by persons skilled in the art that munexous variations 
and/or modifications may be made to the invention as shown in the specific 
embodiments without departing from the spirit or scope of the invention as broadly 
26 described. The present embodiments are, therefore, to be considered in all respects as 
illustristive and not restrictive. 
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CLAIMS: 

I. A method of delivering a test to a candidate^ the test having a plurality of 
questions that are diosen from a plurality of question types ranging from an easiest 
question type to a hardest question type> the method oon^)rising die sequential steps of: 
5 delivering to the candidate a question selected fr<3m the easiest question type 

and, theiieafter, delivering to the candidate one or more questions from the same 
question type; 

delivering to the candidate a question selected from another of the question 
types andy thereaft^, delivering to the candidate one or more questions from the same 
10 question type; 

>vherein at least one question is a '^Swaps'* question, the "Swaps" question 
comprising displaying a plurality of images to flie candidate in a predetermined spatial 
rdationsbip and an instniction sequence associated with the Images for indica 
candidate die required reorganisation of the spatial relationship of the images. 

15 2. A method of delivering a test to a candidate according to claim 1> wherein tibe 
test comprises question types 1. 2, n. where type 1 is the easiest question type and 
type n is the hardest question type, and the method comprises sequentially delivering all 
the questions of a given question type to the candidate and sequMtially delivering to 
the candidate the question types in the ozd^ of question type 1 to question type n. 

20 3* A method of delivering a test to a candidate according to claim 2, wherein the 
candidate is only delivered a question wheal a preceding question in the sequential 
delivery has bo^ answered* 

4. A method of delivering a test to a candidate according to any one of the 
preceding claims^ wherein upon answering a question^ ttie method further comprising 

25 asking the candidate to indicate his or her confidence that the answer to the question 
wascorrecL 

5. A method of delivering a test to a candidate according to claim 4, wherein the 
candidate is asked to indicate his or her confidrace that the answer to the question was 
within one of two or more confidence bands, 

30 6. A method of delivering a test to a candidate according to any one of the 
preceding clauns» wherein the test is delivered by a testing device that comprises a 
display for visually displaying a question to the candidate, an input device for allowing 
the candidate to answer the question and a processor for driving the display and for 
being responsive to the input device for detemaining the answer provided by the 

35 candidate. 
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7. A mettiod of delivering a test to a candidate according to any one of the 
preceding claims* wherein the test is contained as question data on a storage medium . 
and the processor selectively accesses the question data to deliver the questions 
sequentially. 

5 8. A method of delivering a test to a candidate according to claim 7, wherein Ae 
storage medium comprises sequence data representative of the sequence in whidi the 
question data is accessed by ttio processor. 

9. A method of delivering a test to a candidate according to claim 7 or 8, whorein 
the storage medium is controlled by a s^ver that is remote fiom the processor and 

10 which is responsive to predetennined if^ut fiom the processor for allowing the test to 
be deliv^ed to the candidate. 

10. A method of delivering a test to a candidate according to any one of the 
iveceding claims, wherdn the method furdier comprises recording die time taken for 
the candidate to read and answer a question. 

15 11. A method of delivering a test to a candidate according to claim 4 or 5, wherein 
the method further comprises recording the time taken by the candidate to indicate their 
confidence. 

12. A method of delivering a test to a candidate according to any one of the 
preceding claims, wherein the mediod further comprises the step of assessing the 

20 relative cognitive ability^ of the candidate based upon « consideration of any one or 
more of: 

correctness of some or all of the answers provided by the candidate; 
time taken to provide those answers; and 

the relationship between the coiiectness of some or all the answers and 
25 confidence indicated by ihc csndidate that his or her answers to the question was 
correct 

13. An apparatus for delivering a test to a candidate, the test having a plurality of 
questions that arie chosen from a plurality of question types ranging fiom an easiest 
question type to a hardest question type, the e^aratus comprising: 

30 a display for sequentially displaying to the candidate one or more questions 

selected fiom the easiest question type; 

ii^iit means for allowing the candidate to answer the questions and for 

triggering the display to display to the candidate the next question in the sequmce or, if 

at the end of the sequence, one or more questions fiom the next hardest question type; 
35 wherein at least one question is a **Swaps'* question, &e "Swaps" question 

comprising a plurality of images displayed in a predetermined spatial relationship and 
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an instruction sequence associated with the images for indicating to the ca ndi daf fl the 
required reorganisation of the spatial relationship of the images. 

14. An apparatus for delivCTng a test to a candidate according to claim 13 or 14, 
wherein the ii^ut means allows the candidate to indicate his or her confidence diat the 

5 answer to the question was correct 

1 5. An apparatus for delivmng a test to a candidate according to daim 14, wherein 
the input meaiis aUows the candidate to indicate his or her confidence that the answer to 
&e question was wifliin one of two or more confidence bands. 

16. An apparatus for delivering a test to a candidate according to any one of claims 
10 13 to IS, the apparatas further comprises: 

a storage medium on which the test is stored as question data; and 
a processor selectively accesses the question data to deliver the questions 
sequentially. 

17. An apparatus for delivering a test to a candidate according to claim 16, wherein 
15 the storage medium comprises sequence data rqiresentative of the sequence in wliich 

the question data is accessed by the processor. 

18. An apparatus for delivering a test to a candidate according to claim 16 or 17, 
wherein the stmage medium is controlled by a server that is remote from the processor 
and which is responsive to predetermined input from the processor for allowing die test 

20 to be ddivered to ttie candidate. 

19. An ^aratus for delivering a test to a candidate according to any one of claims 
13 to 18» wher»n the apparatus further comprises timing means to record the time taken 
for the candidate to read and answer a question, . 

20. An apparatus for delivering a test to a candidate according to any one of claims 
25 13 to 18, i^erein the ^jparatus comprises a processor for assessing the relative 

y cognitive ability of ttie candidate based upon a consideration of any one or more ot 

correctness of some or all of the answers provided by the candidate; 
time taken to provide those answers; and 

the relationship between the correctness of some or all the answers and 
30 confidence indicated by the candidate that his or her answers to the question was 
correct 

21. A method of assessing the relative cognitive ability of a candidate^ the 
assessment having a plurality of questions that are chosen fiom a plurality of question 
types ranging progressively from an easiest question type to a hardest question type, the 

35 method comprising the sequential steps of: 

delivering to fte candidate a question selected fiom the easiest question type; 
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allowing the candidate to answer the question Bnd» thereafter, delivering to the 
candidate sequentially one or more questions fiom the same question type; 

determining a quantifier in response to the answers provided by the candidate to 
the questions of the easiest question type; 
5 if the quantifier is above a predetermined threshold for deliv^ng to the 

candidate a question selected from another of the question types, delivering to the 
candidate a question selected from the said another of the question types and, thereaifler» 
delivering to the candidate one or more questions from the same question type; and 

deriving an assessment of the relative cognitive ability of a candidate using the 
10 quantifier and the question types for which answers have been provided. 

22. A method of assessing the relative cognitive ability of a candidate according to 
claim 21 » wherein the said another of die question types is the next question type in the 
progressive range of question types. 

23. A meOiodofassessing the relative cognitive ^lity of a candidate accordiiig to 
15 daim 21 or 22^ wherein the method comprises detennining a second quantifier in 

response to the answers provided by the candidate to ttie questions of the said anotiier 
question types. 

24. A method of assessing the relative cognitive ability of a candidate according to 
claim 23, wherein if the second quantifier is above a predetermined second threshold 

20 fixr delivering to the candidate a question selected from tfie next question type in the 
progression of question types, the mediod fiirther comprises the steps of delivering to 
the candidate a question selected from the next question type and tfaereafler, delivering 
to the candidate one or more questions from the same question type. 

25. A method fi}r assessing the relative cognitive ability of a candidate according to 
25 any one of claims 21 to 24, wherein at least one question is a "Swaps" question, the 

"Swaps" question con^yrising a phmdify of images displayed in a predetermined spatial 
relationship and an instruction sequence associated with the images for indicating to the 
candidate the required reorganisation of tiie spatial relationship of the images. 

26. A method or ^aratus according to any one of claims 21 to 24, wheiein the 
3d quantifier is determined upon a consideration of any one or more of: 

correctness of some or all of ttie answers provided by the candidate; 
time taken to provide those answers; and 

the relationship between the correctness of some or all the answers and 
confidence indicated by the candidate that his or her answers to the question was 
35 correct 
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27. An ^aratus for assessing die relative cognitive ability of a candidate, the 
assessmmt having a plviraiity of questions that are chosen from a plurality of question 
types ranging progressively from an easiest question type to a hardest question type, the 
apparatus comprising: 

5 a processor for driving a display to deliver to the candidate a question selected 

from the easiest question type; 

an mpjAt device for allowing tiie candidate to answer the question, wheron the 
processor is responsive to the answer being provided to subsequentiy ddiver to the 
candidate sequentially one or more questions from the same question type; 
10 a calculation module being responsive to tiie answers provided by the candidate 

to the questions of the easiest question type for detennining a quantifier, wherein the 
processor is responsive to the quantifier being above a prcdeteimined threshold for 
driving the display to deliver to tiie candidate a question selected from anoth^ of the 
question Q^pes and, fhereafker, delivering to the candidate one or more questions from 
16 the same question type; and 

an assessment module for being responsive to the quantifier and the question 
types for \diich answers have been provided for deriving an assessment of the relative 
cognitive ability of a candidate. 

28. An fi^aratus for assessing the rdative cognitive ability of a candidate according 
20 to daim 27 , wherein the said another of the question types is tiie next question type in 

the progressive range of question types. 

29. An apparatus for assessing the relative cognitive ability of a candidate according 
to claim 27 or 28, wherein the calculation module fiarther determines a second 
quantifier in response to the answers provided by the candidate to tiie questions of the 

25 said another question types. 

30. An apparatus for assessing the relative cognitive ability of a candidate according 
to claim 29, wherein tiie processor is also responsive to fho second quantifier being 
above a predetennined second threshold for driving tiie display to deliver to the 
candidate a question selected fiom the next question type in the progression of question 

30 types, and thereafter, delivering to the candida[te one or more questions from the same 
question type* 

31. An apparatus for assessing tiie relative cognitive ability of a candidate according 
to any one of claims 27 to 30, wherein fiie calculation module determines tiie quantifier 
based upon any one or more of: 

35 correctness of some or all of the answers provided by the candidate; 

time taken to provide tiiose answers; and 
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the relationsbip betwen ihe coirectness of some or all the answeis and 
confidence indicated by ttie candidate that his or her answers to the question was 
correct. 

32. An apparatus for assessing the relative cognitive ability of a candidate according 
5 to any one of claims 27 to 31, wherein at least one question is a **Swaps" question, the 

"Swaps" question compiising a phirality of images displayed in a predetmnined spatial 
relationship and an instruction sequence associated with the iniages for indicating to the 
candidate the required reorganisation of the spatial relationship of die images. 

33. A method of assessing the progress of a candidate in response to a learning 
10 program, &e assessment having a plurality of questions that are chosen from a plurality 

of question types ranging progressively from an easiest question type to a hardest 
question type^ the method conqprising the sequential steps oft 

delivering to fiie candidate a question selected fiom the easiest question type; 
allowing the candidate to answer the question and, thereafter, ddivering to the 
15 candidate sequentially one or more questions fix»n the same question type in response 
to the preceding question being answered; 

det^mining a quantifier in re^nse to the answers provided by the candidate to 
the questions of the eaidest question type; 

if the quantifi^ is above a predetermined tiueshold for delivering to the 
20 candidate a question selected fi'om another of the question types, delivering to the 
candidate a question selected fiom the said another of the question types and, thereafter, 
delivering to the candidate one or more questions fiom the same question type; and 

deriving an assessment of the relative cognitive ability of a candidate using the 
quantifier and the question types for whidi answers have been provided. 
25 34. A method of assessing the egress of a candidate in response to a learning 
program according to claim 33, wherdn the quantifier is detomined upon a 
consideration of any one or more of: 

conectness of some or all of the answers provided by the candidate; 

time taken to provide those answers; and 
.30 the relationship between tiie conectness of some or all die answers and 

confidence indicated by the candidate that his or her answers to tiie question was 
correct 

35. A method of assessing Che progress of a candidate in response to a learning 
program of any one of claims 33 or 34, wherein at least one question is a ^Swaps" 
35 question, the "Swaps" question comprising displaying a plurality of images to the 
candidate in a spatial relationship and an instruction sequence associated with the 
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images for indicating to the candidate the required reoxganiaation of the spatial 
relationship of the images. 

36. A *'Swaps" question according to any one of claims 1» 13» 25, 32 or 35, wherein 
the question has three images that are dearly visually distinct 
5 37. A "Swaps" question according to any one of claims 1, 13^ 25* 32» 35 or 36» 
whoreln to answer the "Swaps^ question, flie candidate dicks and drags the images to 
the spatial locadons that are believed to be correct. 

38. A "Swaps'* question according to any one of claims 1, 13, 25, 32, 35, 36 or 37, 
wherein a plurality of alternative answers to a "Swaps" question is displayed to the 

1 0 candidate, and the candidate selects one of the alternatives. 

39. A method of delivering a test to a candidate according to any one of clain:is 1 to 
12, whmin. all the questions are "Swaps" questions and the Instruction sequence of 
each question in a particular question type indudes the same number of swap 
instructions requiring the spatial reorganisation of two images. 

15 40. A mediod of delivering a test to a candidate according to claim 39, wherein the 
harder the question type, the larger the number of swaps instructions are induded in the 
instruction sequence. 

41. An Apparatus for delivering a test to a candidate according to any one of daixns 
13 to 20, herein all the questions are "Swaps" questions and the instruction sequence 

20 of eadi question in a particular question type includes the same number of swap 
instructions requiring the spatial reorganisation of two images. 

42. A method of delivering a test to a candidate according to claim 41 wherein the 
harder the question type, the larger the number of swaps instmctioas are included in the 
instrucdon sequence. 

25 43. A method for assessing the relative cognitive ability of a candidate according to 
claim 25. i^erein all the questions are "Swi^s" qMStlons and the instruction sequence 
of eadi question in a particular question type includes the same number of svfBpp 
instructions requiring the spatial reorganisation of two images. 

44. A method of delivering a test to a candidate according to claim 43, wherein the 
30 harder the question type, the larger the number of swl^>8 instructions are induded in the 

instruction sequence. 

45. An apparatus fbr assessing tiie rdative cognitive abiUty of a candidate according 
to claim 32, wherein all the questions are "Swaps" questions and the instruction 
sequence of each qxiestion in a particular question type includes the same number of 

35 sw^> instructions requiring the spatial reorganisation of two images. 
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46* A method of ddivexing a test to a candidate according to claim 45, wherein the 
harder the question type, the larger the number of swaps instructions are included in the 
instruction sequence. 

47. A method for assessing the progress of the candidate in response to a learning 
5 program according to claim 3S, wherein all the questions are "Swaps*^ questions and the 

instruction sequence of each question in a particular question type includes die same 
number of swap instructions requiting the spatial reorganisation of two images. 

48. A method of deliv^g a test to a candidate according to claim 47, wherein the 
harder the question type, the larger the number of swaps instructions are included in the 

1 0 instruction sequence. 
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